Virtual environments are typically employed in electronic games that are played on game consoles, personal computers, and other types of computing devices, as well as in other types of applications. A user usually interacts with entities in a virtual environment by manipulating a mouse, joystick, wheel, game pad, track ball, or other user input device that causes the virtual entity to move in a specific manner or carry out some other action or function as defined by the software program that produces the virtual environment. The effects of a user interaction with an entity in the virtual environment are generally visible on a display. For example, a user might be controlling a virtual entity such as a spaceship or race car that is displayed in the virtual environment. Thus, the interaction between a user and virtual entities in a virtual environment is well known.
Another form of user input employs displays that are responsive to the touch of a user's finger or a stylus. Touch responsive displays can be pressure activated, responsive to electrical capacitance, changes in magnetic field intensity, employ surface acoustic waves, or respond to other variables that indicate the location of a finger or stylus on the display. Another type of touch sensitive display includes a plurality of optical sensors spaced-apart around the periphery of the display screen so that the location of a finger or stylus touching the screen can be detected. Using one of these touch sensitive displays, a user can more directly control a virtual entity or image that is displayed. For example, the user may touch the displayed virtual entity to select it and then drag the entity to a new position on the touch-sensitive display, or touch a control and drag the control to change some parameter.
However, in most such touch-sensitive displays, the response is only to the touch of the finger or stylus at a point. There is another type of interaction with a virtual environment that might provide a much richer experience for the user. While virtual environments produced for example, by electronic game software programs often include virtual entities that are displayed on a screen, it would be desirable for the virtual environment to also respond to physical objects that are placed on the display surface. In most prior art touch-sensitive displays, the finger or stylus is treated simply an alternative type of pointing device used to make selections or drag elements about on the screen. To be truly interactive to physical objects that are placed on it, a display surface should also be able to detect where one or more physical objects are placed on it, as well as detect different types of physical objects, each of which might provide a different interactive experience for the user. However, the capacitive, electromagnetic, optical, or other types of sensors used in conventional touch-sensitive displays typically cannot simultaneously detect the location of more than one finger or object touching the display screen at a time, and thus, would be unable to detect the location or each different type of a plurality of different types of physical objects placed thereon. These prior art touch-sensing systems are generally incapable of detecting more than a point of contact and are unable to detect the shape of an object proximate to or touching the display surface. Even capacitive or resistive, or acoustic surface wave sensing display surfaces that can detect multiple points of contact are unable to image objects placed on a display surface to any reasonable degree of resolution (detail), and most require expensive or relatively complicated coding schemes, than a more desirable simple bar code. Prior art systems of these types cannot detect patterns on an object or detailed shapes that might be used to identify each object among a plurality of different objects that are placed on a display surface.
Another user interface approach that has been developed in the prior art uses cameras mounted to the side and above a horizontal display screen to visually capture an image of a user's finger or other objects that are touching the display screen. This multiple camera mounting configuration is clearly not a compact system that most people would want to use in a residential setting. In addition, the accuracy of this type of multi-camera system in responding to an object that is on or proximate to the display surface depends upon the capability of the software used with the system to visually recognize objects and their location in three-dimensional space. Furthermore, the view of one object by one of the cameras may be blocked by an intervening object. Also it is difficult to deduce if a finger or object has touched the screen, and such a vision sensing system often requires an involved calibration. From an aesthetic viewpoint, objects usable in such a system will not be pleasing to a user because they will need a code that is most likely visible to the user on top of the object, and thus, the manner in which the object is being detected will be clearly evident to the user.
To address many of the problems inherent in the types of touch-sensitive displays discussed above, a user interface platform was developed in the MIT Media Lab, as reported by Brygg Ullmer and Hiroshi Ishii in “The metaDESK: Models and Prototypes for Tangible User Interfaces,” Proceedings of UIST 10/1997:14-17. The metaDESK includes a near-horizontal graphical surface used to display two-dimensional geographical information. Above the graphical surface is disposed an arm-mounted flat-panel display that serves as an “active lens” for use in displaying three dimensional geographical information. A computer vision system inside the desk unit (i.e., below the graphical surface) includes infrared (IR) lamps, an IR camera, a video camera, a video projector, and mirrors. The mirrors reflect the graphical image projected by the projector onto the underside of the graphical display surface. The IR camera can detect passive objects called “phicons” that are placed on the graphical surface. Magnetic-field position sensors and electrical-contact sensors are also included in the metaDESK. For example, in response to the IR camera detecting an IR marking applied to the bottom of a “Great Dome phicon,” a map of the MIT campus is displayed on the graphical surface, with the actual location of the Great Dome in the map positioned where the Great Dome phicon is located. Moving the Great Dome phicon over the graphical surface manipulates the displayed map by rotating or translating the map in correspondence to the movement of the phicon by a user.
A similar approach to sensing objects on a display surface is disclosed in several papers published by Jun Rekimoto of Sony Computer Science Laboratory, Inc. in collaboration with others. These papers briefly describe a “HoloWall” and a “HoloTable,” both of which use IR light to detect objects that are proximate to or in contact with a display surface on which a rear-projected image is visible. The rear-projection panel, which is vertical in the HoloWall and horizontal in the HoloTable, is semi-opaque and diffusive, so that objects become more clearly visible as they approach and then contact the panel. The objects thus detected can be a user's fingers or hands, or other objects.
It would be desirable to enable interaction within a virtual environment between physical objects that are placed on a display surface and virtual objects or images that are displayed. Ideally, the interaction should go well beyond the use of a physical object to simply select and move a virtual object on a touch-sensitive screen, as has been done in the prior art. The interaction between physical objects and virtual objects or images that are displayed might take different forms. For example, it would be desirable to be able to position a physical object on the display surface, have the software program executing the virtual environment recognize the physical object and its location, and then respond to the presence of the physical object when displaying virtual objects. Also, the virtual object or image should be visible through the object and either be displayed in response to the physical object being placed on the display surface or modified when viewed through the physical object as a result of the physical object being detected on the displays surface. Thus, if the user moves a physical object to change its position or orientation on the display surface, in response, the software program producing the virtual entity of image should respond by changing some function, parameter, or feature related to the virtual entity or image. Also, the marking applied to the physical object should be invisible to the user and generally not effect visible light that is transmitted through the object. It would also be desirable to provide different markings on opposed surfaces of a physical object that will be detected when a specific surface of the object is placed on the display surface, so that each side surface of the object can cause different functions to be achieved.
There are many potential applications of such a physical object when used for interacting with a software program. By enabling a user to employ such a physical object in a clearly intuitive manner, the enjoyment and functionality of software programs executing on an interactive surface can be greatly improved.